County Cluster R
This is a rewrite of my County Cluster notebook - originally written in Python in Jupyter Notebook - using R. The code may not be pretty, buy my goal is simply to practice R and demonstrate basic competence.
Loading packages, setting random seed
library(tidyverse)Registered S3 methods overwritten by 'dbplyr':
method from
print.tbl_lazy
print.tbl_sql
── Attaching packages ──────────────────────────────────── tidyverse 1.3.1 ──
✔ ggplot2 3.3.6 ✔ purrr 0.3.4
✔ tibble 3.1.7 ✔ dplyr 1.0.9
✔ tidyr 1.2.0 ✔ stringr 1.4.0
✔ readr 2.1.2 ✔ forcats 0.5.1
── Conflicts ─────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()library(modelr)
library(maps)
Attaching package: ‘maps’
The following object is masked from ‘package:purrr’:
maplibrary(mapproj)
library(htmlwidgets)Registered S3 methods overwritten by 'htmltools':
method from
print.html tools:rstudio
print.shiny.tag tools:rstudio
print.shiny.tag.list tools:rstudio
Registered S3 method overwritten by 'htmlwidgets':
method from
print.htmlwidget tools:rstudioset.seed(123)Read in datasets
ed_data <- read_csv("../data/Education.csv", locale = locale(encoding = "Latin1"))Rows: 3283 Columns: 47
── Column specification ─────────────────────────────────────────────────────
Delimiter: ","
chr (2): State, Area name
dbl (25): FIPS Code, 2003 Rural-urban Continuum Code, 2003 Urban Influenc...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.pop_data <- read_csv("../data/PopulationEstimates.csv", locale = locale(encoding = "Latin1"))Warning: One or more parsing issues, see `problems()` for detailsRows: 3274 Columns: 149
── Column specification ─────────────────────────────────────────────────────
Delimiter: ","
chr (3): FIPS, State, Area_Name
dbl (57): Rural-urban_Continuum Code_2003, Rural-urban_Continuum Code_201...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.pov_data <- read_csv("../data/PovertyEstimates.csv", locale = locale(encoding = "Latin1"))Rows: 3193 Columns: 34
── Column specification ─────────────────────────────────────────────────────
Delimiter: ","
chr (2): Stabr, Area_name
dbl (17): FIPStxt, Rural-urban_Continuum_Code_2003, Urban_Influence_Code_...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.unm_data <- read_csv("../data/Unemployment.csv", locale = locale(encoding = "Latin1"))Rows: 3275 Columns: 56
── Column specification ─────────────────────────────────────────────────────
Delimiter: ","
chr (3): State, Area_name, Median_Household_Income_2018
dbl (17): FIPS, Rural_urban_continuum_code_2013, Urban_influence_code_201...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.Taking a look at the education data
head(ed_data)And the ed_data dimensions
dim(ed_data)[1] 3283 47Now looking at population data - it had a problem when reading in, we’ll have to interrogate that more later
head(pop_data)pop_data dimensions
dim(pop_data)[1] 3274 149And now poverty data
head(pov_data)poverty dimensions
dim(pov_data)[1] 3193 34And finally, unemployment data
head(unm_data)and unemployment data dimensions
dim(unm_data)[1] 3275 56A quick look at the summaries of each
#summary(ed_data)#summary(pop_data)#summary(pov_data)#summary(unm_data)So it turns out that while summary() in R returns much the same information as .describe() does in pandas, it does it in a really ungainly format. I don’t think I’ll be doing that again. I’ve commented out the summaries to avoid clogging the notebook. At this point in pandas I listed out the data types in each dataframe, but here those have already been displayed on the head(). I also loaded in a shape file with the geometries of the counties, but this was ultimately only used for plotting and the maps package in R already covers US county mapping, so I won’t be using the shape file for this.
First, a tiny bit of housekeeping - making all the FIPS columns called just “FIPS”.
ed_data <- rename(ed_data, FIPS = "FIPS Code")
pov_data <- rename(pov_data, FIPS = FIPStxt)Before joining, need to check that FIPS are actually unique
dim(distinct(select(ed_data, FIPS)))[1] == dim(ed_data)[1][1] TRUEdim(distinct(select(pop_data, FIPS)))[1] == dim(pop_data)[1][1] TRUEdim(distinct(select(pov_data, FIPS)))[1] == dim(pov_data)[1][1] TRUEdim(distinct(select(unm_data, FIPS)))[1] == dim(unm_data)[1][1] TRUEAll good on that front. Of the four dataframes, three read in FIPS as dbl while pop_data read it in as chr. While the latter is more technically correct, I’ll convert pop_data FIPS to dbl so all four are on the same page when it comes to leading zeros and data types.
(pop_data <- mutate(pop_data, across(FIPS, as.double)))And now to join them all on FIPS. And yes, the name of big_data is tongue-in-cheek.
big_data <- ed_data %>%
full_join(pop_data, by = c("FIPS")) %>%
full_join(pov_data, by = c("FIPS")) %>%
full_join(unm_data, by = c("FIPS"))Let’s take a look at the result
head(big_data)dim(big_data)[1] 3284 283and now let’s see how many nulls there are
sort(colSums(is.na(big_data))) FIPS
1
State.x
1
Area name
1
State
9
Area_name.y
9
Less than a high school diploma, 2014-18
11
High school diploma only, 2014-18
11
Some college or associate's degree, 2014-18
11
Bachelor's degree or higher, 2014-18
11
Percent of adults with less than a high school diploma, 2014-18
11
Percent of adults with a high school diploma only, 2014-18
11
Percent of adults completing some college or associate's degree, 2014-18
11
Percent of adults with a bachelor's degree or higher, 2014-18
11
State.y
11
Area_Name
11
CENSUS_2010_POP
11
ESTIMATES_BASE_2010
11
POP_ESTIMATE_2010
11
POP_ESTIMATE_2011
11
POP_ESTIMATE_2012
11
POP_ESTIMATE_2013
11
POP_ESTIMATE_2014
11
POP_ESTIMATE_2015
11
POP_ESTIMATE_2016
11
POP_ESTIMATE_2017
11
POP_ESTIMATE_2018
11
Less than a high school diploma, 2000
12
High school diploma only, 2000
12
Some college or associate's degree, 2000
12
Bachelor's degree or higher, 2000
12
Percent of adults with less than a high school diploma, 2000
12
Percent of adults with a high school diploma only, 2000
12
Percent of adults completing some college or associate's degree, 2000
12
Percent of adults with a bachelor's degree or higher, 2000
12
Civilian_labor_force_2010
12
Employed_2010
12
Unemployed_2010
12
Unemployment_rate_2010
12
Civilian_labor_force_2011
12
Employed_2011
12
Unemployed_2011
12
Unemployment_rate_2011
12
Civilian_labor_force_2012
12
Employed_2012
12
Unemployed_2012
12
Unemployment_rate_2012
12
Civilian_labor_force_2013
12
Employed_2013
12
Unemployed_2013
12
Unemployment_rate_2013
12
Civilian_labor_force_2014
12
Employed_2014
12
Unemployed_2014
12
Unemployment_rate_2014
12
Civilian_labor_force_2015
12
Employed_2015
12
Unemployed_2015
12
Unemployment_rate_2015
12
Civilian_labor_force_2016
12
Employed_2016
12
Unemployed_2016
12
Unemployment_rate_2016
12
Civilian_labor_force_2017
12
Employed_2017
12
Unemployed_2017
12
Unemployment_rate_2017
12
Civilian_labor_force_2018
12
Employed_2018
12
Unemployed_2018
12
Unemployment_rate_2018
12
Less than a high school diploma, 1990
13
High school diploma only, 1990
13
Some college or associate's degree, 1990
13
Bachelor's degree or higher, 1990
13
Percent of adults with less than a high school diploma, 1990
13
Percent of adults with a high school diploma only, 1990
13
Percent of adults with a bachelor's degree or higher, 1990
13
Percent of adults completing some college or associate's degree, 1990
14
Civilian_labor_force_2007
14
Employed_2007
14
Unemployed_2007
14
Unemployment_rate_2007
14
Civilian_labor_force_2008
14
Employed_2008
14
Unemployed_2008
14
Unemployment_rate_2008
14
Civilian_labor_force_2009
14
Employed_2009
14
Unemployed_2009
14
Unemployment_rate_2009
14
Less than a high school diploma, 1980
17
High school diploma only, 1980
17
Some college (1-3 years), 1980
17
Four years of college or higher, 1980
17
Percent of adults with less than a high school diploma, 1980
17
Percent of adults with a high school diploma only, 1980
17
Percent of adults completing some college (1-3 years), 1980
17
Percent of adults completing four years of college or higher, 1980
17
Metro_2013
62
2003 Rural-urban Continuum Code
63
2003 Urban Influence Code
63
2013 Rural-urban Continuum Code
63
2013 Urban Influence Code
63
Rural-urban_Continuum Code_2013
64
Urban_Influence_Code_2013.x
64
Rural_urban_continuum_code_2013
65
Urban_influence_code_2013
65
Rural-urban_Continuum Code_2003
69
Urban_Influence_Code_2003.x
69
N_POP_CHG_2010
90
N_POP_CHG_2011
90
N_POP_CHG_2012
90
N_POP_CHG_2013
90
N_POP_CHG_2014
90
N_POP_CHG_2015
90
N_POP_CHG_2016
90
N_POP_CHG_2017
90
N_POP_CHG_2018
90
Births_2010
90
Births_2011
90
Births_2012
90
Births_2013
90
Births_2014
90
Births_2015
90
Births_2016
90
Births_2017
90
Births_2018
90
Deaths_2010
90
Deaths_2011
90
Deaths_2012
90
Deaths_2013
90
Deaths_2014
90
Deaths_2015
90
Deaths_2016
90
Deaths_2017
90
Deaths_2018
90
NATURAL_INC_2010
90
NATURAL_INC_2011
90
NATURAL_INC_2012
90
NATURAL_INC_2013
90
NATURAL_INC_2014
90
NATURAL_INC_2015
90
NATURAL_INC_2016
90
NATURAL_INC_2017
90
NATURAL_INC_2018
90
INTERNATIONAL_MIG_2010
90
INTERNATIONAL_MIG_2011
90
INTERNATIONAL_MIG_2012
90
INTERNATIONAL_MIG_2013
90
INTERNATIONAL_MIG_2014
90
INTERNATIONAL_MIG_2015
90
INTERNATIONAL_MIG_2016
90
INTERNATIONAL_MIG_2017
90
INTERNATIONAL_MIG_2018
90
DOMESTIC_MIG_2010
90
DOMESTIC_MIG_2011
90
DOMESTIC_MIG_2012
90
DOMESTIC_MIG_2013
90
DOMESTIC_MIG_2014
90
DOMESTIC_MIG_2015
90
DOMESTIC_MIG_2016
90
DOMESTIC_MIG_2017
90
DOMESTIC_MIG_2018
90
NET_MIG_2010
90
NET_MIG_2011
90
NET_MIG_2012
90
NET_MIG_2013
90
NET_MIG_2014
90
NET_MIG_2015
90
NET_MIG_2016
90
NET_MIG_2017
90
NET_MIG_2018
90
RESIDUAL_2010
90
RESIDUAL_2011
90
RESIDUAL_2012
90
RESIDUAL_2014
90
RESIDUAL_2015
90
RESIDUAL_2016
90
RESIDUAL_2017
90
RESIDUAL_2018
90
GQ_ESTIMATES_BASE_2010
90
GQ_ESTIMATES_2010
90
GQ_ESTIMATES_2011
90
GQ_ESTIMATES_2012
90
GQ_ESTIMATES_2013
90
GQ_ESTIMATES_2014
90
GQ_ESTIMATES_2015
90
GQ_ESTIMATES_2016
90
GQ_ESTIMATES_2017
90
GQ_ESTIMATES_2018
90
R_birth_2011
91
R_birth_2012
91
R_birth_2013
91
R_birth_2014
91
R_birth_2015
91
R_birth_2016
91
R_birth_2017
91
R_birth_2018
91
R_death_2011
91
R_death_2012
91
R_death_2013
91
R_death_2014
91
R_death_2015
91
R_death_2016
91
R_death_2017
91
R_death_2018
91
R_NATURAL_INC_2011
91
R_NATURAL_INC_2012
91
R_NATURAL_INC_2013
91
R_NATURAL_INC_2014
91
R_NATURAL_INC_2015
91
R_NATURAL_INC_2016
91
R_NATURAL_INC_2017
91
R_NATURAL_INC_2018
91
R_INTERNATIONAL_MIG_2011
91
R_INTERNATIONAL_MIG_2012
91
R_INTERNATIONAL_MIG_2013
91
R_INTERNATIONAL_MIG_2014
91
R_INTERNATIONAL_MIG_2015
91
R_INTERNATIONAL_MIG_2016
91
R_INTERNATIONAL_MIG_2017
91
R_INTERNATIONAL_MIG_2018
91
R_DOMESTIC_MIG_2011
91
R_DOMESTIC_MIG_2012
91
R_DOMESTIC_MIG_2013
91
R_DOMESTIC_MIG_2014
91
R_DOMESTIC_MIG_2015
91
R_DOMESTIC_MIG_2016
91
R_DOMESTIC_MIG_2017
91
R_DOMESTIC_MIG_2018
91
R_NET_MIG_2011
91
R_NET_MIG_2012
91
R_NET_MIG_2013
91
R_NET_MIG_2014
91
R_NET_MIG_2015
91
R_NET_MIG_2016
91
R_NET_MIG_2017
91
R_NET_MIG_2018
91
Stabr
91
Area_name.x
91
POVALL_2018
91
CI90LBAll_2018
91
CI90UBALL_2018
91
PCTPOVALL_2018
91
CI90LBALLP_2018
91
CI90UBALLP_2018
91
POV017_2018
91
CI90LB017_2018
91
CI90UB017_2018
91
PCTPOV017_2018
91
CI90LB017P_2018
91
CI90UB017P_2018
91
POV517_2018
91
CI90LB517_2018
91
CI90UB517_2018
91
PCTPOV517_2018
91
CI90LB517P_2018
91
CI90UB517P_2018
91
MEDHHINC_2018
91
CI90LBINC_2018
91
CI90UBINC_2018
91
Median_Household_Income_2018
91
RESIDUAL_2013
92
Med_HH_Income_Percent_of_State_Total_2018
92
Less than a high school diploma, 1970
98
High school diploma only, 1970
98
Some college (1-3 years), 1970
98
Four years of college or higher, 1970
98
Percent of adults with less than a high school diploma, 1970
98
Percent of adults with a high school diploma only, 1970
98
Percent of adults completing some college (1-3 years), 1970
98
Percent of adults completing four years of college or higher, 1970
98
Economic_typology_2015
142
Rural-urban_Continuum_Code_2013
143
Urban_Influence_Code_2013.y
143
Rural-urban_Continuum_Code_2003
148
Urban_Influence_Code_2003.y
148
POV04_2018
3232
CI90LB04_2018
3232
CI90UB04_2018
3232
PCTPOV04_2018
3232
CI90LB04P_2018
3232
CI90UB04P_2018
3232 All columns have at least one null (I suspect this is due to the parsing problem from pop_data), a large number have <20, another significant group have 50-100, and a handful have 140-150 or outright majority nulls. On closer inspection, the outright majority null columns are state-level data which is not necessary for my analysis. Many of the other nulls may be the result of state-levels rows. Before anything else, I’ll save a copy of the full data
write_csv(big_data, "../data/outer_join_R.csv")And now to clean it up. First to take out the mostly null columns
(big_data <- select(big_data, -c(POV04_2018, CI90LB04_2018, CI90UB04_2018, PCTPOV04_2018, CI90LB04P_2018, CI90UB04P_2018)))Now the number of rows with nulls is much more reasonable
sum(!complete.cases(big_data))[1] 159Let’s see if my suspicion there’s a fully null row is correct
filter(big_data, apply(is.na(big_data), 1, all))Getting rid of the fully null row
big_data <- filter(big_data, !apply(is.na(big_data), 1, all))Now let’s look at the remaining rows with at least one null
filter(big_data, apply(is.na(big_data), 1, any))That’s mostly Puerto Rico and Alaska, as well as the summary rows for states and the entire country. Here’s the rows with at least one null which DON’T meet any of the criteria I just listed.
filter(big_data, apply(is.na(big_data), 1, any) & !(State.x %in% c("AK", "PR")) & (FIPS %% 1000 != 0))At this point in the original pandas notebook I broke to review external research to better understand what was going on with these counties (and county equivalents). I decided to narrow down to just columns I thought I might actually use before cleaning further, as the time component complicated cleaning some counties pretty severely.
(big_slice <- select(big_data, FIPS, State.x, "Area name", '2013 Rural-urban Continuum Code', '2013 Urban Influence Code', "Percent of adults with less than a high school diploma, 2014-18", 'Percent of adults with a high school diploma only, 2014-18', "Percent of adults completing some college or associate's degree, 2014-18", "Percent of adults with a bachelor's degree or higher, 2014-18", 'Economic_typology_2015', 'POP_ESTIMATE_2010', 'POP_ESTIMATE_2018', 'R_NATURAL_INC_2018', 'R_NET_MIG_2018', 'PCTPOVALL_2018', 'Unemployment_rate_2010', 'Unemployment_rate_2018', 'Median_Household_Income_2018'))NANow I’ll get rid of Alaska, Hawaii, Puerto Rico, and the summary rows
(big_slice <- filter(big_slice, !(State.x %in% c("AK", "PR", "HI")) & (FIPS %% 1000 != 0)))Now let’s see what nulls remain
filter(big_slice, apply(is.na(big_slice), 1, any))Based on external research, I think I can drop all three of these without losing much
(big_slice <- drop_na(big_slice))Now to finish the cleaning. I’ll make FIPS an int in R instead of a chr for mapping reasons which will come up later
(big_slice <- mutate(big_slice, across(FIPS, as.integer)))The Rural-urban codes should be ints as well
(big_slice <- mutate(big_slice, across("2013 Rural-urban Continuum Code":"2013 Urban Influence Code", as.integer)))Economic typology should really be a chr
(big_slice <- mutate(big_slice, across(Economic_typology_2015, as.character)))Population estimates are currently doubles but should be integers
(big_slice <- mutate(big_slice, across(POP_ESTIMATE_2010:POP_ESTIMATE_2018, as.integer)))And last but not least for type changes, median household income is currently a chr with dollar signs and thousand separators, neither of which we want
big_slice$Median_Household_Income_2018 <- str_replace_all(big_slice$Median_Household_Income_2018, "[[:punct:]/$]", "")(big_slice <- mutate(big_slice, across(Median_Household_Income_2018, as.integer)))Much better. Now I can create some new variables
(big_slice <- mutate(big_slice,
Unemployment_change = Unemployment_rate_2018 - Unemployment_rate_2010,
Population_percent_change = (POP_ESTIMATE_2018 - POP_ESTIMATE_2010) / POP_ESTIMATE_2010
))And also one-hot-encode the Economic Typology
big_slice %>%
mutate(ones = 1) %>%
pivot_wider(
names_from = Economic_typology_2015,
values_from = ones,
names_prefix = "Econ_Typology_",
names_sort = TRUE,
values_fill = 0,
values_fn = as.integer)Saving a copy of the modeling-ready data
write_csv(big_slice, "../data/processed_R.csv")Let’s specify the X and y to use for modeling - more features may be used later
(X <- select(big_slice, 6, 13, 14, 17, 18, 19, 20))y <- big_slice$PCTPOVALL_2018Train test split
train_len <- as.integer(0.75 * dim(X)[1])
train_test_split <- sample(c(rep(0, train_len), rep(1, dim(X)[1] - train_len)))
X_train <- X[train_test_split == 0, ]
X_test <- X[train_test_split == 1, ]
y_train <- y[train_test_split == 0]
y_test <- y[train_test_split == 1]Scaling the data
X_train_sc <- scale(X_train)Can we get that scale back?
train_scale <- attributes(X_train_sc)$`scaled:scale`
train_centers <- attributes(X_train_sc)$`scaled:center`And now to apply that to X_test
X_test_sc <- as_tibble(scale(X_test, center = train_centers, scale = train_scale))
X_sc <- as_tibble(scale(X, center = train_centers, scale = train_scale))
X_train_sc <- as_tibble(X_train_sc)Now a linear model
linreg <- lm(y_train ~ Unemployment_rate_2018 + `Percent of adults with less than a high school diploma, 2014-18` + R_NATURAL_INC_2018 + R_NET_MIG_2018 + Median_Household_Income_2018 + Unemployment_change + Population_percent_change, data = X_train_sc)Looking at the model summary - note that the R2 isn’t great, but not bad at all
summary(linreg)
Call:
lm(formula = y_train ~ Unemployment_rate_2018 + `Percent of adults with less than a high school diploma, 2014-18` +
R_NATURAL_INC_2018 + R_NET_MIG_2018 + Median_Household_Income_2018 +
Unemployment_change + Population_percent_change, data = X_train_sc)
Residuals:
Min 1Q Median 3Q Max
-16.0391 -1.8902 -0.4046 1.3001 24.0312
Coefficients:
Estimate
(Intercept) 15.19936
Unemployment_rate_2018 1.26757
`Percent of adults with less than a high school diploma, 2014-18` 1.69539
R_NATURAL_INC_2018 0.28640
R_NET_MIG_2018 -0.60838
Median_Household_Income_2018 -3.43256
Unemployment_change -0.11910
Population_percent_change 0.43550
Std. Error
(Intercept) 0.06619
Unemployment_rate_2018 0.07794
`Percent of adults with less than a high school diploma, 2014-18` 0.08450
R_NATURAL_INC_2018 0.08785
R_NET_MIG_2018 0.09590
Median_Household_Income_2018 0.09375
Unemployment_change 0.07581
Population_percent_change 0.11117
t value
(Intercept) 229.619
Unemployment_rate_2018 16.263
`Percent of adults with less than a high school diploma, 2014-18` 20.063
R_NATURAL_INC_2018 3.260
R_NET_MIG_2018 -6.344
Median_Household_Income_2018 -36.613
Unemployment_change -1.571
Population_percent_change 3.917
Pr(>|t|)
(Intercept) < 2e-16
Unemployment_rate_2018 < 2e-16
`Percent of adults with less than a high school diploma, 2014-18` < 2e-16
R_NATURAL_INC_2018 0.00113
R_NET_MIG_2018 2.68e-10
Median_Household_Income_2018 < 2e-16
Unemployment_change 0.11630
Population_percent_change 9.21e-05
(Intercept) ***
Unemployment_rate_2018 ***
`Percent of adults with less than a high school diploma, 2014-18` ***
R_NATURAL_INC_2018 **
R_NET_MIG_2018 ***
Median_Household_Income_2018 ***
Unemployment_change
Population_percent_change ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 3.196 on 2323 degrees of freedom
Multiple R-squared: 0.7271, Adjusted R-squared: 0.7262
F-statistic: 884 on 7 and 2323 DF, p-value: < 2.2e-16And the sorted coefs. NB: I didn’t sort by absolute value here, so they aren’t in order of magnitude of effect.
sort(coef(linreg)) Median_Household_Income_2018
-3.4325582
R_NET_MIG_2018
-0.6083822
Unemployment_change
-0.1191027
R_NATURAL_INC_2018
0.2864007
Population_percent_change
0.4354999
Unemployment_rate_2018
1.2675679
`Percent of adults with less than a high school diploma, 2014-18`
1.6953941
(Intercept)
15.1993565 How does it perform on the test set? Manually calculating Adj. R2 on test set
1 - (sum((y_test - add_predictions(X_test_sc, linreg)$pred)^2) / (dim(X_test_sc)[1] - dim(X_test_sc)[2] - 1)) / (sum((y_test - mean(y_test))^2) / (dim(X_test_sc)[1] - 1))[1] 0.7005477Now to get predictions on the entire scaled dataset (for mapping)
X_sc <- add_predictions(X_sc, linreg)Mapping the error
palette(rainbow(50, start = 0, end = 0.3, rev = TRUE))
offset <- abs(min(X_sc$pred))
regions <- inner_join(county.fips, mutate(big_slice, pred = X_sc$pred), by = c("fips" = "FIPS"))
map('county', region = regions$polyname, exact=TRUE, fill = TRUE, col = (regions$pred + offset), lwd = 0.2)
FIPS code 46102 refers to Oglala Lakota County, which doesn’t appear in this fips directory due to a name change from Shannon County in 2014 which also changed the FIPS code.
filter(county.fips, polyname == "south dakota,shannon" | fips == 46102)Plotting the predictions against actual to visually evaluate the model. I’d love to use a diverging palette centered on 0 here, but I’m moving on for the sake of time.
ggplot(data = X_sc) +
geom_point(mapping = aes(x = y, y = pred, color = (y - pred)))
Now let’s look at a model accounting for interactions
linreg2 <- lm(y_train ~ Unemployment_rate_2018 * `Percent of adults with less than a high school diploma, 2014-18` * R_NATURAL_INC_2018 * R_NET_MIG_2018 * Median_Household_Income_2018 * Unemployment_change * Population_percent_change, data = X_train_sc)
#summary(linreg2)So that summary ended up being kind of huge, but hey! Significantly higher R^2 values and still a ridiculously small p-value. Good stuff. Summary commented out for cleanliness. Mapping the error for this one - still need to get better at working with palettes to have this be on the same scale as the previous map.
regions2 <- inner_join(county.fips, mutate(big_slice, pred = add_predictions(X_sc, linreg2)$pred), by = c("fips" = "FIPS"))
offset2 <- abs(min(regions$pred))
map('county', region = regions2$polyname, exact=TRUE, fill = TRUE, col = (regions$pred + offset2), lwd = 0.2)
And the test R^2 for linreg2
1 - (sum((y_test - add_predictions(X_test_sc, linreg2)$pred)^2) / (dim(X_test_sc)[1] - dim(X_test_sc)[2] - 1)) / (sum((y_test - mean(y_test))^2) / (dim(X_test_sc)[1] - 1))[1] 0.7716487Testing R^2 has improved as well, though not by as much, which is to be expected. Plotting the predicted against actual again for the interaction mode. Same caveat with the color palette as the similar plot above.
ggplot(data = add_predictions(X_sc, linreg2)) +
geom_point(mapping = aes(x = y, y = pred, color = (y - pred)))
And plotting error against actual
ggplot(data = add_predictions(X_sc, linreg2)) +
geom_point(alpha = 0.2, mapping = aes(x = y, y = (y - pred))) +
geom_ref_line(h = 0, colour = "red")
The errors visibly curve upwards at higher values. Investigating some of the worst errors. Still getting the hang of pipes.
big_slice %>% mutate( ErrorPoly = add_predictions(X_sc, linreg2)$pred - y) %>% arrange(desc(ErrorPoly))Worth noting that if I did this correctly (not an insignificant “if”) my highest errors are quite different than the original Python ones. Strange and worth looking into.
Now the main event - Kmeans! Using the Lloyd algorithm to match with the version I created in sklearn.
(clustering <- kmeans(select(X_sc, !pred), centers = 8, iter.max = 1000, nstart = 100, algorithm = "Lloyd"))K-means clustering with 8 clusters of sizes 540, 197, 555, 479, 194, 691, 200, 252
Cluster means:
Percent of adults with less than a high school diploma, 2014-18
1 0.50434023
2 1.55167945
3 0.02795914
4 -0.74836368
5 -0.32018264
6 -0.39873646
7 1.43060433
8 -0.93149162
R_NATURAL_INC_2018 R_NET_MIG_2018 Unemployment_rate_2018
1 -0.6263823 0.3595697 0.3104808
2 -0.2309517 -0.8658475 2.1137316
3 -0.6755405 -0.4238099 0.5229862
4 0.0676130 -0.6335739 -0.8609722
5 0.7051512 1.8549927 -0.4078850
6 0.1316757 0.3582573 -0.2998739
7 1.5197683 -0.7205129 -0.1399658
8 0.8441239 0.2856057 -0.6806658
Median_Household_Income_2018 Unemployment_change Population_percent_change
1 -0.6308310 -1.2286659 -0.21441392
2 -1.1651410 -0.5631025 -0.84830246
3 -0.5067658 0.2504223 -0.64959653
4 0.1106185 1.1713470 -0.42662892
5 0.8471009 -0.1438088 2.32917032
6 0.2096499 -0.0819266 0.26733050
7 -0.3345929 0.5519030 0.08145902
8 2.1740270 0.3825758 0.87786462
Clustering vector:
[1] 6 5 2 1 1 2 1 3 1 1 1 1 2 1 1 6 1 1 1 3 1 1 3 2 1 6 1 1 1 7 1 2 1 1 6
[36] 1 6 1 3 1 5 5 2 2 6 1 1 6 3 1 6 1 2 1 3 1 1 6 8 2 1 1 6 1 2 2 1 2 3 6
[71] 1 6 6 1 5 1 2 6 5 2 6 2 1 3 1 5 6 1 6 6 2 3 3 3 3 3 3 6 6 3 3 3 3 3 6
[106] 3 1 6 6 6 7 6 3 6 3 2 3 7 3 3 2 3 3 3 6 7 1 4 2 3 3 3 3 3 3 2 3 3 3 6
[141] 3 4 1 2 5 3 3 7 7 1 3 3 1 5 3 3 7 8 3 1 1 1 2 8 1 8 2 2 6 2 6 2 2 1 1
[176] 7 2 8 1 6 2 2 6 7 8 6 8 8 1 5 6 5 7 8 8 1 6 8 8 8 8 1 1 1 8 8 1 2 1 1
[211] 2 1 8 6 1 5 6 8 5 4 3 8 8 5 4 8 3 1 3 5 1 5 1 8 8 8 5 1 8 8 8 5 6 1 6
[246] 8 6 4 6 8 5 1 6 4 6 5 6 6 6 7 3 8 5 4 8 7 6 4 3 8 5 5 8 4 8 6 4 5 4 8
[281] 6 6 8 6 6 8 6 6 6 5 8 6 6 6 1 6 6 1 5 1 5 5 6 1 1 6 6 5 3 1 1 1 1 1 7
[316] 7 1 1 5 1 5 3 1 1 5 5 6 1 6 1 5 1 6 6 4 5 5 1 5 5 6 5 6 5 1 5 5 5 5 6
[351] 5 1 1 3 1 6 5 1 1 1 7 2 1 6 5 6 2 1 1 1 1 3 5 6 2 1 3 6 1 6 6 1 6 7 1
[386] 5 6 2 6 7 8 1 7 5 1 5 1 1 3 5 1 6 1 1 1 6 1 7 5 1 1 7 1 8 1 5 1 8 1 3
[421] 6 1 1 1 8 1 5 2 1 8 1 1 5 6 1 5 1 1 2 1 1 6 1 7 1 6 7 1 5 6 6 1 1 2 6
[456] 2 1 3 2 6 1 6 2 6 6 5 1 5 1 6 6 6 1 3 1 2 1 2 7 6 1 1 1 1 1 1 2 3 1 7
[491] 2 2 2 3 1 2 1 2 1 1 2 5 1 1 5 1 1 1 1 2 2 6 7 1 1 1 1 5 1 6 6 1 6 8 5
[526] 5 8 6 4 6 5 6 7 7 1 6 6 8 6 6 7 1 5 7 5 6 6 3 7 7 7 4 6 7 6 7 1 8 6 5
[561] 6 4 2 3 6 4 3 3 3 3 6 3 3 3 6 3 6 3 6 6 3 7 8 3 3 6 3 3 3 3 3 3 8 3 3
[596] 2 3 3 3 3 6 3 3 3 2 8 3 8 3 8 3 3 3 3 3 3 8 4 3 3 6 3 3 3 3 4 3 8 3 3
[631] 6 6 3 3 4 3 3 2 6 3 3 3 3 3 4 3 3 6 3 3 3 3 3 6 4 4 3 3 3 8 3 1 4 7 6
[666] 6 6 1 8 6 6 1 6 6 6 1 7 6 6 6 1 6 6 1 6 1 6 6 4 1 3 8 8 6 8 1 1 6 6 6
[701] 6 6 1 8 4 6 7 3 1 1 1 6 6 6 1 6 6 6 1 1 1 1 1 6 6 6 6 6 3 6 1 6 1 6 1
[736] 6 6 1 6 3 6 6 6 3 6 1 6 1 6 8 1 1 6 6 6 4 4 6 6 4 4 4 4 8 4 7 4 4 4 4
[771] 4 4 4 4 6 4 4 4 7 5 7 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 6 4 4 4 4 6 6
[806] 6 8 4 4 4 3 8 4 4 4 8 4 6 4 4 4 4 4 4 6 4 4 4 4 4 4 8 4 4 4 4 6 4 8 4
[841] 4 4 4 4 6 8 4 4 4 4 4 6 4 4 3 4 3 4 4 3 4 6 4 3 3 4 4 4 4 4 4 4 6 4 4
[876] 6 6 4 3 4 4 7 7 6 7 4 3 7 7 4 3 7 3 4 7 4 4 4 4 8 7 4 4 3 4 8 4 3 4 4
[911] 4 4 4 4 8 4 3 6 4 4 3 4 4 6 4 4 4 4 8 4 4 4 4 4 4 4 4 4 4 4 6 7 4 4 4
[946] 4 4 4 7 4 4 4 4 4 4 4 3 3 7 1 1 6 3 1 2 2 8 6 3 6 3 2 1 6 1 3 6 6 3 1
[981] 2 1 7 6 2 2 3 1 6 1 2 1 6 1 2 6 2 1 1 6
[ reached getOption("max.print") -- omitted 2108 entries ]
Within cluster sum of squares by cluster:
[1] 1576.3545 1099.2593 1408.4598 1144.5249 1262.7983 1552.7751 1084.8146
[8] 893.7738
(between_SS / total_SS = 53.2 %)
Available components:
[1] "cluster" "centers" "totss" "withinss"
[5] "tot.withinss" "betweenss" "size" "iter"
[9] "ifault" Checking the attributes
attributes(clustering)$names
[1] "cluster" "centers" "totss" "withinss"
[5] "tot.withinss" "betweenss" "size" "iter"
[9] "ifault"
$class
[1] "kmeans"attributes(clustering)$centersNULLI am still confused by attributes in R…
LS0tCnRpdGxlOiAiQ291bnR5IENsdXN0ZXIgUiIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKVGhpcyBpcyBhIHJld3JpdGUgb2YgbXkgQ291bnR5IENsdXN0ZXIgbm90ZWJvb2sgLSBvcmlnaW5hbGx5IHdyaXR0ZW4gaW4gUHl0aG9uIGluIEp1cHl0ZXIgTm90ZWJvb2sgLSB1c2luZyBSLiBUaGUgY29kZSBtYXkgbm90IGJlIHByZXR0eSwgYnV5IG15IGdvYWwgaXMgc2ltcGx5IHRvIHByYWN0aWNlIFIgYW5kIGRlbW9uc3RyYXRlIGJhc2ljIGNvbXBldGVuY2UuIAoKTG9hZGluZyBwYWNrYWdlcywgc2V0dGluZyByYW5kb20gc2VlZApgYGB7cn0KbGlicmFyeSh0aWR5dmVyc2UpCmxpYnJhcnkobW9kZWxyKQpsaWJyYXJ5KG1hcHMpCmxpYnJhcnkobWFwcHJvaikKbGlicmFyeShodG1sd2lkZ2V0cykKc2V0LnNlZWQoMTIzKQpgYGAKClJlYWQgaW4gZGF0YXNldHMKYGBge3J9CmVkX2RhdGEgPC0gcmVhZF9jc3YoIi4uL2RhdGEvRWR1Y2F0aW9uLmNzdiIsIGxvY2FsZSA9IGxvY2FsZShlbmNvZGluZyA9ICJMYXRpbjEiKSkKcG9wX2RhdGEgPC0gcmVhZF9jc3YoIi4uL2RhdGEvUG9wdWxhdGlvbkVzdGltYXRlcy5jc3YiLCBsb2NhbGUgPSBsb2NhbGUoZW5jb2RpbmcgPSAiTGF0aW4xIikpCnBvdl9kYXRhIDwtIHJlYWRfY3N2KCIuLi9kYXRhL1BvdmVydHlFc3RpbWF0ZXMuY3N2IiwgbG9jYWxlID0gbG9jYWxlKGVuY29kaW5nID0gIkxhdGluMSIpKQp1bm1fZGF0YSA8LSByZWFkX2NzdigiLi4vZGF0YS9VbmVtcGxveW1lbnQuY3N2IiwgbG9jYWxlID0gbG9jYWxlKGVuY29kaW5nID0gIkxhdGluMSIpKQpgYGAKVGFraW5nIGEgbG9vayBhdCB0aGUgZWR1Y2F0aW9uIGRhdGEKYGBge3J9CmhlYWQoZWRfZGF0YSkKYGBgCkFuZCB0aGUgZWRfZGF0YSBkaW1lbnNpb25zCmBgYHtyfQpkaW0oZWRfZGF0YSkKYGBgCk5vdyBsb29raW5nIGF0IHBvcHVsYXRpb24gZGF0YSAtIGl0IGhhZCBhIHByb2JsZW0gd2hlbiByZWFkaW5nIGluLCB3ZSdsbCBoYXZlIHRvIGludGVycm9nYXRlIHRoYXQgbW9yZSBsYXRlcgpgYGB7cn0KaGVhZChwb3BfZGF0YSkKYGBgCnBvcF9kYXRhIGRpbWVuc2lvbnMKYGBge3J9CmRpbShwb3BfZGF0YSkKYGBgCkFuZCBub3cgcG92ZXJ0eSBkYXRhCmBgYHtyfQpoZWFkKHBvdl9kYXRhKQpgYGAKcG92ZXJ0eSBkaW1lbnNpb25zCmBgYHtyfQpkaW0ocG92X2RhdGEpCmBgYApBbmQgZmluYWxseSwgdW5lbXBsb3ltZW50IGRhdGEKYGBge3J9CmhlYWQodW5tX2RhdGEpCmBgYAphbmQgdW5lbXBsb3ltZW50IGRhdGEgZGltZW5zaW9ucwpgYGB7cn0KZGltKHVubV9kYXRhKQpgYGAKQSBxdWljayBsb29rIGF0IHRoZSBzdW1tYXJpZXMgb2YgZWFjaApgYGB7cn0KI3N1bW1hcnkoZWRfZGF0YSkKYGBgCmBgYHtyfQojc3VtbWFyeShwb3BfZGF0YSkKYGBgCmBgYHtyfQojc3VtbWFyeShwb3ZfZGF0YSkKYGBgCmBgYHtyfQojc3VtbWFyeSh1bm1fZGF0YSkKYGBgClNvIGl0IHR1cm5zIG91dCB0aGF0IHdoaWxlIHN1bW1hcnkoKSBpbiBSIHJldHVybnMgbXVjaCB0aGUgc2FtZSBpbmZvcm1hdGlvbiBhcyAuZGVzY3JpYmUoKSBkb2VzIGluIHBhbmRhcywgaXQgZG9lcyBpdCBpbiBhIHJlYWxseSB1bmdhaW5seSBmb3JtYXQuIEkgZG9uJ3QgdGhpbmsgSSdsbCBiZSBkb2luZyB0aGF0IGFnYWluLiBJJ3ZlIGNvbW1lbnRlZCBvdXQgdGhlIHN1bW1hcmllcyB0byBhdm9pZCBjbG9nZ2luZyB0aGUgbm90ZWJvb2suIApBdCB0aGlzIHBvaW50IGluIHBhbmRhcyBJIGxpc3RlZCBvdXQgdGhlIGRhdGEgdHlwZXMgaW4gZWFjaCBkYXRhZnJhbWUsIGJ1dCBoZXJlIHRob3NlIGhhdmUgYWxyZWFkeSBiZWVuIGRpc3BsYXllZCBvbiB0aGUgaGVhZCgpLiBJIGFsc28gbG9hZGVkIGluIGEgc2hhcGUgZmlsZSB3aXRoIHRoZSBnZW9tZXRyaWVzIG9mIHRoZSBjb3VudGllcywgYnV0IHRoaXMgd2FzIHVsdGltYXRlbHkgb25seSB1c2VkIGZvciBwbG90dGluZyBhbmQgdGhlIG1hcHMgcGFja2FnZSBpbiBSIGFscmVhZHkgY292ZXJzIFVTIGNvdW50eSBtYXBwaW5nLCBzbyBJIHdvbid0IGJlIHVzaW5nIHRoZSBzaGFwZSBmaWxlIGZvciB0aGlzLiAKCkZpcnN0LCBhIHRpbnkgYml0IG9mIGhvdXNla2VlcGluZyAtIG1ha2luZyBhbGwgdGhlIEZJUFMgY29sdW1ucyBjYWxsZWQganVzdCAiRklQUyIuCmBgYHtyfQplZF9kYXRhIDwtIHJlbmFtZShlZF9kYXRhLCBGSVBTID0gIkZJUFMgQ29kZSIpCnBvdl9kYXRhIDwtIHJlbmFtZShwb3ZfZGF0YSwgRklQUyA9IEZJUFN0eHQpCmBgYAoKQmVmb3JlIGpvaW5pbmcsIG5lZWQgdG8gY2hlY2sgdGhhdCBGSVBTIGFyZSBhY3R1YWxseSB1bmlxdWUKYGBge3J9CmRpbShkaXN0aW5jdChzZWxlY3QoZWRfZGF0YSwgRklQUykpKVsxXSA9PSBkaW0oZWRfZGF0YSlbMV0KYGBgCmBgYHtyfQpkaW0oZGlzdGluY3Qoc2VsZWN0KHBvcF9kYXRhLCBGSVBTKSkpWzFdID09IGRpbShwb3BfZGF0YSlbMV0KYGBgCmBgYHtyfQpkaW0oZGlzdGluY3Qoc2VsZWN0KHBvdl9kYXRhLCBGSVBTKSkpWzFdID09IGRpbShwb3ZfZGF0YSlbMV0KYGBgCmBgYHtyfQpkaW0oZGlzdGluY3Qoc2VsZWN0KHVubV9kYXRhLCBGSVBTKSkpWzFdID09IGRpbSh1bm1fZGF0YSlbMV0KYGBgCkFsbCBnb29kIG9uIHRoYXQgZnJvbnQuIE9mIHRoZSBmb3VyIGRhdGFmcmFtZXMsIHRocmVlIHJlYWQgaW4gRklQUyBhcyBkYmwgd2hpbGUgcG9wX2RhdGEgcmVhZCBpdCBpbiBhcyBjaHIuIFdoaWxlIHRoZSBsYXR0ZXIgaXMgbW9yZSB0ZWNobmljYWxseSBjb3JyZWN0LCBJJ2xsIGNvbnZlcnQgcG9wX2RhdGEgRklQUyB0byBkYmwgc28gYWxsIGZvdXIgYXJlIG9uIHRoZSBzYW1lIHBhZ2Ugd2hlbiBpdCBjb21lcyB0byBsZWFkaW5nIHplcm9zIGFuZCBkYXRhIHR5cGVzLiAKYGBge3J9Cihwb3BfZGF0YSA8LSBtdXRhdGUocG9wX2RhdGEsIGFjcm9zcyhGSVBTLCBhcy5kb3VibGUpKSkKYGBgCkFuZCBub3cgdG8gam9pbiB0aGVtIGFsbCBvbiBGSVBTLiBBbmQgeWVzLCB0aGUgbmFtZSBvZiBiaWdfZGF0YSBpcyB0b25ndWUtaW4tY2hlZWsuCmBgYHtyfQpiaWdfZGF0YSA8LSBlZF9kYXRhICU+JQogIGZ1bGxfam9pbihwb3BfZGF0YSwgYnkgPSBjKCJGSVBTIikpICU+JQogIGZ1bGxfam9pbihwb3ZfZGF0YSwgYnkgPSBjKCJGSVBTIikpICU+JQogIGZ1bGxfam9pbih1bm1fZGF0YSwgYnkgPSBjKCJGSVBTIikpCmBgYApMZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgcmVzdWx0CmBgYHtyfQpoZWFkKGJpZ19kYXRhKQpgYGAKYGBge3J9CmRpbShiaWdfZGF0YSkKYGBgCmFuZCBub3cgbGV0J3Mgc2VlIGhvdyBtYW55IG51bGxzIHRoZXJlIGFyZQpgYGB7cn0Kc29ydChjb2xTdW1zKGlzLm5hKGJpZ19kYXRhKSkpCmBgYApBbGwgY29sdW1ucyBoYXZlIGF0IGxlYXN0IG9uZSBudWxsIChJIHN1c3BlY3QgdGhpcyBpcyBkdWUgdG8gdGhlIHBhcnNpbmcgcHJvYmxlbSBmcm9tIHBvcF9kYXRhKSwgYSBsYXJnZSBudW1iZXIgaGF2ZSA8MjAsIGFub3RoZXIgc2lnbmlmaWNhbnQgZ3JvdXAgaGF2ZSA1MC0xMDAsIGFuZCBhIGhhbmRmdWwgaGF2ZSAxNDAtMTUwIG9yIG91dHJpZ2h0IG1ham9yaXR5IG51bGxzLiBPbiBjbG9zZXIgaW5zcGVjdGlvbiwgdGhlIG91dHJpZ2h0IG1ham9yaXR5IG51bGwgY29sdW1ucyBhcmUgc3RhdGUtbGV2ZWwgZGF0YSB3aGljaCBpcyBub3QgbmVjZXNzYXJ5IGZvciBteSBhbmFseXNpcy4gTWFueSBvZiB0aGUgb3RoZXIgbnVsbHMgbWF5IGJlIHRoZSByZXN1bHQgb2Ygc3RhdGUtbGV2ZWxzIHJvd3MuCkJlZm9yZSBhbnl0aGluZyBlbHNlLCBJJ2xsIHNhdmUgYSBjb3B5IG9mIHRoZSBmdWxsIGRhdGEKYGBge3J9CndyaXRlX2NzdihiaWdfZGF0YSwgIi4uL2RhdGEvb3V0ZXJfam9pbl9SLmNzdiIpCmBgYApBbmQgbm93IHRvIGNsZWFuIGl0IHVwLiBGaXJzdCB0byB0YWtlIG91dCB0aGUgbW9zdGx5IG51bGwgY29sdW1ucwpgYGB7cn0KKGJpZ19kYXRhIDwtIHNlbGVjdChiaWdfZGF0YSwgLWMoUE9WMDRfMjAxOCwgQ0k5MExCMDRfMjAxOCwgQ0k5MFVCMDRfMjAxOCwgUENUUE9WMDRfMjAxOCwgQ0k5MExCMDRQXzIwMTgsIENJOTBVQjA0UF8yMDE4KSkpCmBgYApOb3cgdGhlIG51bWJlciBvZiByb3dzIHdpdGggbnVsbHMgaXMgbXVjaCBtb3JlIHJlYXNvbmFibGUKYGBge3J9CnN1bSghY29tcGxldGUuY2FzZXMoYmlnX2RhdGEpKQpgYGAKTGV0J3Mgc2VlIGlmIG15IHN1c3BpY2lvbiB0aGVyZSdzIGEgZnVsbHkgbnVsbCByb3cgaXMgY29ycmVjdApgYGB7cn0KZmlsdGVyKGJpZ19kYXRhLCBhcHBseShpcy5uYShiaWdfZGF0YSksIDEsIGFsbCkpCmBgYApHZXR0aW5nIHJpZCBvZiB0aGUgZnVsbHkgbnVsbCByb3cKYGBge3J9CmJpZ19kYXRhIDwtIGZpbHRlcihiaWdfZGF0YSwgIWFwcGx5KGlzLm5hKGJpZ19kYXRhKSwgMSwgYWxsKSkKYGBgCk5vdyBsZXQncyBsb29rIGF0IHRoZSByZW1haW5pbmcgcm93cyB3aXRoIGF0IGxlYXN0IG9uZSBudWxsCmBgYHtyfQpmaWx0ZXIoYmlnX2RhdGEsIGFwcGx5KGlzLm5hKGJpZ19kYXRhKSwgMSwgYW55KSkKYGBgClRoYXQncyBtb3N0bHkgUHVlcnRvIFJpY28gYW5kIEFsYXNrYSwgYXMgd2VsbCBhcyB0aGUgc3VtbWFyeSByb3dzIGZvciBzdGF0ZXMgYW5kIHRoZSBlbnRpcmUgY291bnRyeS4gCkhlcmUncyB0aGUgcm93cyB3aXRoIGF0IGxlYXN0IG9uZSBudWxsIHdoaWNoIERPTidUIG1lZXQgYW55IG9mIHRoZSBjcml0ZXJpYSBJIGp1c3QgbGlzdGVkLgpgYGB7cn0KZmlsdGVyKGJpZ19kYXRhLCBhcHBseShpcy5uYShiaWdfZGF0YSksIDEsIGFueSkgJiAhKFN0YXRlLnggJWluJSBjKCJBSyIsICJQUiIpKSAmIChGSVBTICUlIDEwMDAgIT0gMCkpCmBgYApBdCB0aGlzIHBvaW50IGluIHRoZSBvcmlnaW5hbCBwYW5kYXMgbm90ZWJvb2sgSSBicm9rZSB0byByZXZpZXcgZXh0ZXJuYWwgcmVzZWFyY2ggdG8gYmV0dGVyIHVuZGVyc3RhbmQgd2hhdCB3YXMgZ29pbmcgb24gd2l0aCB0aGVzZSBjb3VudGllcyAoYW5kIGNvdW50eSBlcXVpdmFsZW50cykuIEkgZGVjaWRlZCB0byBuYXJyb3cgZG93biB0byBqdXN0IGNvbHVtbnMgSSB0aG91Z2h0IEkgbWlnaHQgYWN0dWFsbHkgdXNlIGJlZm9yZSBjbGVhbmluZyBmdXJ0aGVyLCBhcyB0aGUgdGltZSBjb21wb25lbnQgY29tcGxpY2F0ZWQgY2xlYW5pbmcgc29tZSBjb3VudGllcyBwcmV0dHkgc2V2ZXJlbHkuIApgYGB7cn0KKGJpZ19zbGljZSA8LSBzZWxlY3QoYmlnX2RhdGEsIEZJUFMsIFN0YXRlLngsICJBcmVhIG5hbWUiLCAnMjAxMyBSdXJhbC11cmJhbiBDb250aW51dW0gQ29kZScsICcyMDEzIFVyYmFuIEluZmx1ZW5jZSBDb2RlJywgIlBlcmNlbnQgb2YgYWR1bHRzIHdpdGggbGVzcyB0aGFuIGEgaGlnaCBzY2hvb2wgZGlwbG9tYSwgMjAxNC0xOCIsICdQZXJjZW50IG9mIGFkdWx0cyB3aXRoIGEgaGlnaCBzY2hvb2wgZGlwbG9tYSBvbmx5LCAyMDE0LTE4JywgIlBlcmNlbnQgb2YgYWR1bHRzIGNvbXBsZXRpbmcgc29tZSBjb2xsZWdlIG9yIGFzc29jaWF0ZSdzIGRlZ3JlZSwgMjAxNC0xOCIsICJQZXJjZW50IG9mIGFkdWx0cyB3aXRoIGEgYmFjaGVsb3IncyBkZWdyZWUgb3IgaGlnaGVyLCAyMDE0LTE4IiwgJ0Vjb25vbWljX3R5cG9sb2d5XzIwMTUnLCAnUE9QX0VTVElNQVRFXzIwMTAnLCAnUE9QX0VTVElNQVRFXzIwMTgnLCAnUl9OQVRVUkFMX0lOQ18yMDE4JywgJ1JfTkVUX01JR18yMDE4JywgJ1BDVFBPVkFMTF8yMDE4JywgJ1VuZW1wbG95bWVudF9yYXRlXzIwMTAnLCAnVW5lbXBsb3ltZW50X3JhdGVfMjAxOCcsICdNZWRpYW5fSG91c2Vob2xkX0luY29tZV8yMDE4JykpCiAgICAKYGBgCk5vdyBJJ2xsIGdldCByaWQgb2YgQWxhc2thLCBIYXdhaWksIFB1ZXJ0byBSaWNvLCBhbmQgdGhlIHN1bW1hcnkgcm93cwpgYGB7cn0KKGJpZ19zbGljZSA8LSBmaWx0ZXIoYmlnX3NsaWNlLCAhKFN0YXRlLnggJWluJSBjKCJBSyIsICJQUiIsICJISSIpKSAmIChGSVBTICUlIDEwMDAgIT0gMCkpKQpgYGAKTm93IGxldCdzIHNlZSB3aGF0IG51bGxzIHJlbWFpbgpgYGB7cn0KZmlsdGVyKGJpZ19zbGljZSwgYXBwbHkoaXMubmEoYmlnX3NsaWNlKSwgMSwgYW55KSkKYGBgCkJhc2VkIG9uIGV4dGVybmFsIHJlc2VhcmNoLCBJIHRoaW5rIEkgY2FuIGRyb3AgYWxsIHRocmVlIG9mIHRoZXNlIHdpdGhvdXQgbG9zaW5nIG11Y2gKYGBge3J9CihiaWdfc2xpY2UgPC0gZHJvcF9uYShiaWdfc2xpY2UpKQpgYGAKTm93IHRvIGZpbmlzaCB0aGUgY2xlYW5pbmcuIEknbGwgbWFrZSBGSVBTIGFuIGludCBpbiBSIGluc3RlYWQgb2YgYSBjaHIgZm9yIG1hcHBpbmcgcmVhc29ucyB3aGljaCB3aWxsIGNvbWUgdXAgbGF0ZXIKYGBge3J9CihiaWdfc2xpY2UgPC0gbXV0YXRlKGJpZ19zbGljZSwgYWNyb3NzKEZJUFMsIGFzLmludGVnZXIpKSkKYGBgClRoZSBSdXJhbC11cmJhbiBjb2RlcyBzaG91bGQgYmUgaW50cyBhcyB3ZWxsCmBgYHtyfQooYmlnX3NsaWNlIDwtIG11dGF0ZShiaWdfc2xpY2UsIGFjcm9zcygiMjAxMyBSdXJhbC11cmJhbiBDb250aW51dW0gQ29kZSI6IjIwMTMgVXJiYW4gSW5mbHVlbmNlIENvZGUiLCBhcy5pbnRlZ2VyKSkpCmBgYApFY29ub21pYyB0eXBvbG9neSBzaG91bGQgcmVhbGx5IGJlIGEgY2hyCmBgYHtyfQooYmlnX3NsaWNlIDwtIG11dGF0ZShiaWdfc2xpY2UsIGFjcm9zcyhFY29ub21pY190eXBvbG9neV8yMDE1LCBhcy5jaGFyYWN0ZXIpKSkKYGBgClBvcHVsYXRpb24gZXN0aW1hdGVzIGFyZSBjdXJyZW50bHkgZG91YmxlcyBidXQgc2hvdWxkIGJlIGludGVnZXJzCmBgYHtyfQooYmlnX3NsaWNlIDwtIG11dGF0ZShiaWdfc2xpY2UsIGFjcm9zcyhQT1BfRVNUSU1BVEVfMjAxMDpQT1BfRVNUSU1BVEVfMjAxOCwgYXMuaW50ZWdlcikpKQpgYGAKQW5kIGxhc3QgYnV0IG5vdCBsZWFzdCBmb3IgdHlwZSBjaGFuZ2VzLCBtZWRpYW4gaG91c2Vob2xkIGluY29tZSBpcyBjdXJyZW50bHkgYSBjaHIgd2l0aCBkb2xsYXIgc2lnbnMgYW5kIHRob3VzYW5kIHNlcGFyYXRvcnMsIG5laXRoZXIgb2Ygd2hpY2ggd2Ugd2FudApgYGB7cn0KYmlnX3NsaWNlJE1lZGlhbl9Ib3VzZWhvbGRfSW5jb21lXzIwMTggPC0gc3RyX3JlcGxhY2VfYWxsKGJpZ19zbGljZSRNZWRpYW5fSG91c2Vob2xkX0luY29tZV8yMDE4LCAiW1s6cHVuY3Q6XS8kXSIsICIiKQpgYGAKYGBge3J9CihiaWdfc2xpY2UgPC0gbXV0YXRlKGJpZ19zbGljZSwgYWNyb3NzKE1lZGlhbl9Ib3VzZWhvbGRfSW5jb21lXzIwMTgsIGFzLmludGVnZXIpKSkKYGBgCk11Y2ggYmV0dGVyLiBOb3cgSSBjYW4gY3JlYXRlIHNvbWUgbmV3IHZhcmlhYmxlcwpgYGB7cn0KKGJpZ19zbGljZSA8LSBtdXRhdGUoYmlnX3NsaWNlLAogIFVuZW1wbG95bWVudF9jaGFuZ2UgPSBVbmVtcGxveW1lbnRfcmF0ZV8yMDE4IC0gVW5lbXBsb3ltZW50X3JhdGVfMjAxMCwKICBQb3B1bGF0aW9uX3BlcmNlbnRfY2hhbmdlID0gKFBPUF9FU1RJTUFURV8yMDE4IC0gUE9QX0VTVElNQVRFXzIwMTApIC8gUE9QX0VTVElNQVRFXzIwMTAKKSkKYGBgCkFuZCBhbHNvIG9uZS1ob3QtZW5jb2RlIHRoZSBFY29ub21pYyBUeXBvbG9neQpgYGB7cn0KYmlnX3NsaWNlICU+JSAKICBtdXRhdGUob25lcyA9IDEpICU+JSAKICBwaXZvdF93aWRlcigKICBuYW1lc19mcm9tID0gRWNvbm9taWNfdHlwb2xvZ3lfMjAxNSwKICB2YWx1ZXNfZnJvbSA9IG9uZXMsCiAgbmFtZXNfcHJlZml4ID0gIkVjb25fVHlwb2xvZ3lfIiwKICBuYW1lc19zb3J0ID0gVFJVRSwKICB2YWx1ZXNfZmlsbCA9IDAsCiAgdmFsdWVzX2ZuID0gYXMuaW50ZWdlcikKYGBgClNhdmluZyBhIGNvcHkgb2YgdGhlIG1vZGVsaW5nLXJlYWR5IGRhdGEKYGBge3J9CndyaXRlX2NzdihiaWdfc2xpY2UsICIuLi9kYXRhL3Byb2Nlc3NlZF9SLmNzdiIpCmBgYApMZXQncyBzcGVjaWZ5IHRoZSBYIGFuZCB5IHRvIHVzZSBmb3IgbW9kZWxpbmcgLSBtb3JlIGZlYXR1cmVzIG1heSBiZSB1c2VkIGxhdGVyCmBgYHtyfQooWCA8LSBzZWxlY3QoYmlnX3NsaWNlLCA2LCAxMywgMTQsIDE3LCAxOCwgMTksIDIwKSkKeSA8LSBiaWdfc2xpY2UkUENUUE9WQUxMXzIwMTgKYGBgClRyYWluIHRlc3Qgc3BsaXQKYGBge3J9CnRyYWluX2xlbiA8LSBhcy5pbnRlZ2VyKDAuNzUgKiBkaW0oWClbMV0pCnRyYWluX3Rlc3Rfc3BsaXQgPC0gc2FtcGxlKGMocmVwKDAsIHRyYWluX2xlbiksIHJlcCgxLCBkaW0oWClbMV0gLSB0cmFpbl9sZW4pKSkKWF90cmFpbiA8LSBYW3RyYWluX3Rlc3Rfc3BsaXQgPT0gMCwgXQpYX3Rlc3QgPC0gWFt0cmFpbl90ZXN0X3NwbGl0ID09IDEsIF0KeV90cmFpbiA8LSB5W3RyYWluX3Rlc3Rfc3BsaXQgPT0gMF0KeV90ZXN0IDwtIHlbdHJhaW5fdGVzdF9zcGxpdCA9PSAxXQpgYGAKU2NhbGluZyB0aGUgZGF0YQpgYGB7cn0KWF90cmFpbl9zYyA8LSBzY2FsZShYX3RyYWluKQpgYGAKQ2FuIHdlIGdldCB0aGF0IHNjYWxlIGJhY2s/CmBgYHtyfQp0cmFpbl9zY2FsZSA8LSBhdHRyaWJ1dGVzKFhfdHJhaW5fc2MpJGBzY2FsZWQ6c2NhbGVgCnRyYWluX2NlbnRlcnMgPC0gYXR0cmlidXRlcyhYX3RyYWluX3NjKSRgc2NhbGVkOmNlbnRlcmAKYGBgCkFuZCBub3cgdG8gYXBwbHkgdGhhdCB0byBYX3Rlc3QKYGBge3J9ClhfdGVzdF9zYyA8LSBhc190aWJibGUoc2NhbGUoWF90ZXN0LCBjZW50ZXIgPSB0cmFpbl9jZW50ZXJzLCBzY2FsZSA9IHRyYWluX3NjYWxlKSkKWF9zYyA8LSBhc190aWJibGUoc2NhbGUoWCwgY2VudGVyID0gdHJhaW5fY2VudGVycywgc2NhbGUgPSB0cmFpbl9zY2FsZSkpClhfdHJhaW5fc2MgPC0gYXNfdGliYmxlKFhfdHJhaW5fc2MpCmBgYApOb3cgYSBsaW5lYXIgbW9kZWwKYGBge3J9CmxpbnJlZyA8LSBsbSh5X3RyYWluIH4gVW5lbXBsb3ltZW50X3JhdGVfMjAxOCArIGBQZXJjZW50IG9mIGFkdWx0cyB3aXRoIGxlc3MgdGhhbiBhIGhpZ2ggc2Nob29sIGRpcGxvbWEsIDIwMTQtMThgICsgUl9OQVRVUkFMX0lOQ18yMDE4ICsgUl9ORVRfTUlHXzIwMTggKyBNZWRpYW5fSG91c2Vob2xkX0luY29tZV8yMDE4ICsgVW5lbXBsb3ltZW50X2NoYW5nZSArIFBvcHVsYXRpb25fcGVyY2VudF9jaGFuZ2UsIGRhdGEgPSBYX3RyYWluX3NjKQpgYGAKTG9va2luZyBhdCB0aGUgbW9kZWwgc3VtbWFyeSAtIG5vdGUgdGhhdCB0aGUgUjIgaXNuJ3QgZ3JlYXQsIGJ1dCBub3QgYmFkIGF0IGFsbApgYGB7cn0Kc3VtbWFyeShsaW5yZWcpCmBgYApBbmQgdGhlIHNvcnRlZCBjb2Vmcy4gTkI6IEkgZGlkbid0IHNvcnQgYnkgYWJzb2x1dGUgdmFsdWUgaGVyZSwgc28gdGhleSBhcmVuJ3QgaW4gb3JkZXIgb2YgbWFnbml0dWRlIG9mIGVmZmVjdC4KYGBge3J9CnNvcnQoY29lZihsaW5yZWcpKQpgYGAKSG93IGRvZXMgaXQgcGVyZm9ybSBvbiB0aGUgdGVzdCBzZXQ/IE1hbnVhbGx5IGNhbGN1bGF0aW5nIEFkai4gUjIgb24gdGVzdCBzZXQKYGBge3J9CjEgLSAoc3VtKCh5X3Rlc3QgLSBhZGRfcHJlZGljdGlvbnMoWF90ZXN0X3NjLCBsaW5yZWcpJHByZWQpXjIpIC8gKGRpbShYX3Rlc3Rfc2MpWzFdIC0gZGltKFhfdGVzdF9zYylbMl0gLSAxKSkgLyAoc3VtKCh5X3Rlc3QgLSBtZWFuKHlfdGVzdCkpXjIpIC8gKGRpbShYX3Rlc3Rfc2MpWzFdIC0gMSkpCmBgYApOb3cgdG8gZ2V0IHByZWRpY3Rpb25zIG9uIHRoZSBlbnRpcmUgc2NhbGVkIGRhdGFzZXQgKGZvciBtYXBwaW5nKQpgYGB7cn0KWF9zYyA8LSBhZGRfcHJlZGljdGlvbnMoWF9zYywgbGlucmVnKQpgYGAKTWFwcGluZyB0aGUgZXJyb3IKYGBge3J9CnBhbGV0dGUocmFpbmJvdyg1MCwgc3RhcnQgPSAwLCBlbmQgPSAwLjMsIHJldiA9IFRSVUUpKQpvZmZzZXQgPC0gYWJzKG1pbihYX3NjJHByZWQpKQpyZWdpb25zIDwtIGlubmVyX2pvaW4oY291bnR5LmZpcHMsIG11dGF0ZShiaWdfc2xpY2UsIHByZWQgPSBYX3NjJHByZWQpLCBieSA9IGMoImZpcHMiID0gIkZJUFMiKSkKbWFwKCdjb3VudHknLCByZWdpb24gPSByZWdpb25zJHBvbHluYW1lLCBleGFjdD1UUlVFLCBmaWxsID0gVFJVRSwgY29sID0gKHJlZ2lvbnMkcHJlZCArIG9mZnNldCksIGx3ZCA9IDAuMikKYGBgCkZJUFMgY29kZSA0NjEwMiByZWZlcnMgdG8gT2dsYWxhIExha290YSBDb3VudHksIHdoaWNoIGRvZXNuJ3QgYXBwZWFyIGluIHRoaXMgZmlwcyBkaXJlY3RvcnkgZHVlIHRvIGEgbmFtZSBjaGFuZ2UgZnJvbSBTaGFubm9uIENvdW50eSBpbiAyMDE0IHdoaWNoIGFsc28gY2hhbmdlZCB0aGUgRklQUyBjb2RlLiAKYGBge3J9CmZpbHRlcihjb3VudHkuZmlwcywgcG9seW5hbWUgPT0gInNvdXRoIGRha290YSxzaGFubm9uIiB8IGZpcHMgPT0gNDYxMDIpCmBgYAoKUGxvdHRpbmcgdGhlIHByZWRpY3Rpb25zIGFnYWluc3QgYWN0dWFsIHRvIHZpc3VhbGx5IGV2YWx1YXRlIHRoZSBtb2RlbC4gSSdkIGxvdmUgdG8gdXNlIGEgZGl2ZXJnaW5nIHBhbGV0dGUgY2VudGVyZWQgb24gMCBoZXJlLCBidXQgSSdtIG1vdmluZyBvbiBmb3IgdGhlIHNha2Ugb2YgdGltZS4gCmBgYHtyfQpnZ3Bsb3QoZGF0YSA9IFhfc2MpICsgCiAgZ2VvbV9wb2ludChtYXBwaW5nID0gYWVzKHggPSB5LCB5ID0gcHJlZCwgY29sb3IgPSAoeSAtIHByZWQpKSkKYGBgCk5vdyBsZXQncyBsb29rIGF0IGEgbW9kZWwgYWNjb3VudGluZyBmb3IgaW50ZXJhY3Rpb25zCmBgYHtyfQpsaW5yZWcyIDwtIGxtKHlfdHJhaW4gfiBVbmVtcGxveW1lbnRfcmF0ZV8yMDE4ICogYFBlcmNlbnQgb2YgYWR1bHRzIHdpdGggbGVzcyB0aGFuIGEgaGlnaCBzY2hvb2wgZGlwbG9tYSwgMjAxNC0xOGAgKiBSX05BVFVSQUxfSU5DXzIwMTggKiBSX05FVF9NSUdfMjAxOCAqIE1lZGlhbl9Ib3VzZWhvbGRfSW5jb21lXzIwMTggKiBVbmVtcGxveW1lbnRfY2hhbmdlICogUG9wdWxhdGlvbl9wZXJjZW50X2NoYW5nZSwgZGF0YSA9IFhfdHJhaW5fc2MpCiNzdW1tYXJ5KGxpbnJlZzIpCmBgYApTbyB0aGF0IHN1bW1hcnkgZW5kZWQgdXAgYmVpbmcga2luZCBvZiBodWdlLCBidXQgaGV5ISBTaWduaWZpY2FudGx5IGhpZ2hlciBSXjIgdmFsdWVzIGFuZCBzdGlsbCBhIHJpZGljdWxvdXNseSBzbWFsbCBwLXZhbHVlLiBHb29kIHN0dWZmLiBTdW1tYXJ5IGNvbW1lbnRlZCBvdXQgZm9yIGNsZWFubGluZXNzLgpNYXBwaW5nIHRoZSBlcnJvciBmb3IgdGhpcyBvbmUgLSBzdGlsbCBuZWVkIHRvIGdldCBiZXR0ZXIgYXQgd29ya2luZyB3aXRoIHBhbGV0dGVzIHRvIGhhdmUgdGhpcyBiZSBvbiB0aGUgc2FtZSBzY2FsZSBhcyB0aGUgcHJldmlvdXMgbWFwLiAKYGBge3J9CnJlZ2lvbnMyIDwtIGlubmVyX2pvaW4oY291bnR5LmZpcHMsIG11dGF0ZShiaWdfc2xpY2UsIHByZWQgPSBhZGRfcHJlZGljdGlvbnMoWF9zYywgbGlucmVnMikkcHJlZCksIGJ5ID0gYygiZmlwcyIgPSAiRklQUyIpKQpvZmZzZXQyIDwtIGFicyhtaW4ocmVnaW9ucyRwcmVkKSkKbWFwKCdjb3VudHknLCByZWdpb24gPSByZWdpb25zMiRwb2x5bmFtZSwgZXhhY3Q9VFJVRSwgZmlsbCA9IFRSVUUsIGNvbCA9IChyZWdpb25zJHByZWQgKyBvZmZzZXQyKSwgbHdkID0gMC4yKQpgYGAKQW5kIHRoZSB0ZXN0IFJeMiBmb3IgbGlucmVnMgpgYGB7cn0KMSAtIChzdW0oKHlfdGVzdCAtIGFkZF9wcmVkaWN0aW9ucyhYX3Rlc3Rfc2MsIGxpbnJlZzIpJHByZWQpXjIpIC8gKGRpbShYX3Rlc3Rfc2MpWzFdIC0gZGltKFhfdGVzdF9zYylbMl0gLSAxKSkgLyAoc3VtKCh5X3Rlc3QgLSBtZWFuKHlfdGVzdCkpXjIpIC8gKGRpbShYX3Rlc3Rfc2MpWzFdIC0gMSkpCmBgYApUZXN0aW5nIFJeMiBoYXMgaW1wcm92ZWQgYXMgd2VsbCwgdGhvdWdoIG5vdCBieSBhcyBtdWNoLCB3aGljaCBpcyB0byBiZSBleHBlY3RlZC4gUGxvdHRpbmcgdGhlIHByZWRpY3RlZCBhZ2FpbnN0IGFjdHVhbCBhZ2FpbiBmb3IgdGhlIGludGVyYWN0aW9uIG1vZGUuIFNhbWUgY2F2ZWF0IHdpdGggdGhlIGNvbG9yIHBhbGV0dGUgYXMgdGhlIHNpbWlsYXIgcGxvdCBhYm92ZS4KYGBge3J9CmdncGxvdChkYXRhID0gYWRkX3ByZWRpY3Rpb25zKFhfc2MsIGxpbnJlZzIpKSArIAogIGdlb21fcG9pbnQobWFwcGluZyA9IGFlcyh4ID0geSwgeSA9IHByZWQsIGNvbG9yID0gKHkgLSBwcmVkKSkpCmBgYApBbmQgcGxvdHRpbmcgZXJyb3IgYWdhaW5zdCBhY3R1YWwKYGBge3J9CmdncGxvdChkYXRhID0gYWRkX3ByZWRpY3Rpb25zKFhfc2MsIGxpbnJlZzIpKSArIAogIGdlb21fcG9pbnQoYWxwaGEgPSAwLjIsIG1hcHBpbmcgPSBhZXMoeCA9IHksIHkgPSAoeSAtIHByZWQpKSkgKyAKICBnZW9tX3JlZl9saW5lKGggPSAwLCBjb2xvdXIgPSAicmVkIikKYGBgClRoZSBlcnJvcnMgdmlzaWJseSBjdXJ2ZSB1cHdhcmRzIGF0IGhpZ2hlciB2YWx1ZXMuIApJbnZlc3RpZ2F0aW5nIHNvbWUgb2YgdGhlIHdvcnN0IGVycm9ycy4gU3RpbGwgZ2V0dGluZyB0aGUgaGFuZyBvZiBwaXBlcy4gCmBgYHtyfQpiaWdfc2xpY2UgJT4lIG11dGF0ZSggRXJyb3JQb2x5ID0gYWRkX3ByZWRpY3Rpb25zKFhfc2MsIGxpbnJlZzIpJHByZWQgLSB5KSAlPiUgYXJyYW5nZShkZXNjKEVycm9yUG9seSkpCmBgYApXb3J0aCBub3RpbmcgdGhhdCBpZiBJIGRpZCB0aGlzIGNvcnJlY3RseSAobm90IGFuIGluc2lnbmlmaWNhbnQgImlmIikgbXkgaGlnaGVzdCBlcnJvcnMgYXJlIHF1aXRlIGRpZmZlcmVudCB0aGFuIHRoZSBvcmlnaW5hbCBQeXRob24gb25lcy4gU3RyYW5nZSBhbmQgd29ydGggbG9va2luZyBpbnRvLiAKCk5vdyB0aGUgbWFpbiBldmVudCAtIEttZWFucyEgVXNpbmcgdGhlIExsb3lkIGFsZ29yaXRobSB0byBtYXRjaCB3aXRoIHRoZSB2ZXJzaW9uIEkgY3JlYXRlZCBpbiBza2xlYXJuLiAKYGBge3J9CihjbHVzdGVyaW5nIDwtIGttZWFucyhzZWxlY3QoWF9zYywgIXByZWQpLCBjZW50ZXJzID0gOCwgaXRlci5tYXggPSAxMDAwLCBuc3RhcnQgPSAxMDAsIGFsZ29yaXRobSA9ICJMbG95ZCIpKQpgYGAKQ2hlY2tpbmcgdGhlIGF0dHJpYnV0ZXMKYGBge3J9CmF0dHJpYnV0ZXMoY2x1c3RlcmluZykKYGBgCmBgYHtyfQphdHRyaWJ1dGVzKGNsdXN0ZXJpbmcpJGNlbnRlcnMKYGBgCkkgYW0gc3RpbGwgY29uZnVzZWQgYnkgYXR0cmlidXRlcyBpbiBSLi4u